Applying Context-Based Prediction in Adversarial Watkins’ Q(λ)-Learning

This paper exhibits the transformation of Watkins’ Q(λ) learning algorithm into an adversarial Qlearning algorithm. A method called context-based prediction, borrowed from multimedia data coding, is used as opponent modeling and is incorporated in the transformed, CBQ(λ) algorithm. We tested CBQ(λ) by playing against three opponents. The first opponent had no prior knowledge and discovered policies as it played. The second opponent carried prior knowledge and used a fixed policy. The third opponent carried prior knowledge and continued to improve its prior policy as play progressed. CBQ(λ) performed well by playing tic-tac-toe against the three opponents listed previously, when given 10 seconds of simulation time.